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_  ABSTRACT 

V 

A  method  is  described  of  extracting  linear  features  from 
images.  The  approach  is  to  construct  a  series  of  lower-resolu¬ 
tion  versions  of  the  original  image  (a  pyramid) ,  and  to  look 
for  lines  in  these  images.  A  line  in  a  low-resolution  image 
corresponds  to  a  thicker  linear  feature  in  a  high-resolution 
image.  The  position  and  extent  of  this  linear  feature  is 
calculated  from  the  low-resolution  image,  and  a  threshold  is 
found  which,  when  applied  in  the  neighborhood  of  the  feature 
in  the  high-resolution  image,  segments  the  linear  feature  from 
its  background.  Advantages  of  the  method  are  that  only  the 
parts  of  the  image  in  the  neighborhood  of  linear  features  need 
be  thresholded,  and  that  different  thresholds  may  be  used  to 
extract  the  various  linear  features  in  the  image. 
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1.  Introduction 

Linear  features  in  aerial  photographs  are  usually  strong 
evidence  of  human  activity,  and  can  also  be  useful  for 
navigation  purposes.  Linear  features  also  occur  in  other  kinds 
of  images,  such  as  bubble  chamber  photographs  and  industrial 
radiographs.  This  paper  presents  a  method  of  finding  those 
parts  of  an  image  that  correspond  to  linear  features,  and  of 
segmenting  these  features  from  their  background. 

The  method  involves  constructing  a  series  of  successively 
lower-resolution  images  (a  pyramid)  from  the  original  image 
(Hanson  and  Riseman,  1978,  Tamimoto,  1978,  Uhr,  1978).  A  line- 


detector  is  run  over  each  of  the  low-resolution  images,  and  its 
output  is  used  to  calculate  regions  in  the  original  image  that 
correspond  to  linear  features.  Lines  detected  in  successively 
lower  resolution  images  correspond  to  successively  thicker  linear 
features  in  the  original  image.  Properties  of  the  regions  in 
the  low-resolution  image  that  give  rise  to  line-detector  responses 
are  used  to  calculate  a  threshold  for  extracting  the  linear  fea¬ 
tures  in  the  original  image.  This  work  is  an  extension  of  an 
earlier  system  for  detecting  blob- like  objects  using  pyramids 


(Shneier,  1979). 
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2.  The  Algorithm 


The  algorithm  performs  two  tasks.  First,  it  finds  the 
parts  of  the  image  that  correspond  to  linear  features,  and  then 
it  calculates  a  threshold  to  extract  these  features.  Both 
tasks  make  use  of  the  pyramid  structure. 

1.  If  the  whole  pyramid  has  been  constructed,  stop.  Other¬ 
wise,  read  in  the  previous  pyramid  level  (the  picture,  if 
this  is  the  first  iteration) . 

2.  Build  a  new  level. 

3.  Apply  a  line-detector  to  the  new  level  (see  below). 

(4.  Run  a  line  enhancement  algorithm  on  the  new  level). 

5.  Group  the  responses  from  steps  3  and  4  into  lines  (see 

below) . 

6.  For  each  line 

a.  calculate  a  threshold  (see  below) 

b.  apply  the  threshold  to  the  region  in  the  original 
image  corresponding  to  the  line  and  write  the  results 
to  the  output  picture. 

7.  Go  to  1. 

A  pyramid  level  is  constructed  from  its  predecessor  by  re¬ 
placing  each  two-by-two  neighborhood  of  points  by  their  median 
value.  (It  is  also  possible  to  use  the  average  value,  but  this 
results  in  more  blurring.)  Thus,  each  successive  level  is  one- 
quarter  the  size  of  the  previous  level. 


The  line  detector  that  is  run  over  the  new  low-resolution 
image  was  designed  to  take  the  neighborhood  size  into  account. 

It  has  to  be  able  to  deal  with  lines  that  are  two  pixels  wide, 
because  it  is  possible  that  such  lines  might  otherwise  "fall 
between  the  cracks"  when  the  next  level  is  constructed.  That 

is,  one  part  of  the  line  might  become  part  of  a  different  neigh¬ 
borhood  than  the  other  part,  and  the  resulting  points  might  no 
longer  be  detectable  as  belonging  to  a  line.  For  horizontal 

and  vertical  lines,  a  three  by  five  pixel  mask  is  used  (Figure  7) , 
while  for  diagonal  lines,  a  five  by  five  mask  is  used  (Figure  2) . 

Each  mask  is  applied  at  every  point  in  the  image,  and  the 
final  value  for  the  point  is  calculated  as  a  function  of  the 
responses.  Thus,  if  two  masks  respond  (for  example,  the  horizon¬ 
tal  and  45°  masks)  an  interpolated  value  is  calculated  as  a 
weighted  average  of  the  responses,  usually  resulting  in  a  direc¬ 
tion  somewhere  between  those  of  the  two  masks  (e.g.  30  degrees) . 
The  responses  are  all  reduced  modulo  180°  to  enable  simpler 
tests  for  similarity  to  be  made  in  the  enhancement  phase  described 
below.  Each  point  has  a  magnitude  and  direction  associated  with 

it.  Figure  3  shows  an  example  of  the  output  of  the  line  detector 
applied  to  a  picture  of  part  of  an  airport  (Cumberland  Municipal 
Airport  in  Maryland) . 

An  optional,  but  usually  advisable,  step  in  the  algorithm 
is  the  application  of  a  line  enhancement  program  to  the  raw 


i 

i 

line-detector  output.  The  purpose  of  this  step  is  to  remove 
noise  from  the  line-detector  output,  and  to  adjust  the  direc¬ 
tions  of  neighboring  points  to  make  them  more  compatible.  The 
enhancement  program  that  was  used  was  adapted  from  that  described 
by  Peleg  (1978) . 

For  each  neighbor  Q  of  a  point  P,  let  Mq  and  Dq  be  the  mag¬ 
nitude  and  direction  of  the  line  response  at  Q;  let  Dp  be  the 
direction  of  the  response  at  P,  and  let  D  be  the  direction  of 
the  line  joining  P  to  Q.  Then  the  degree  to  which  the  response 
at  Q  supports  that  at  P  is 

SQ  =  Mq  cos (Dq— D) cos (Dp— D) 

Note  that  this  is  a  maximum  value  when  Dp  and  Dq  are  both 
collinear  with  D.  Using  this  function,  new  estimates  of  the 
magnitude  and  direction  of  the  edge  at  P  can  be  computed  as 
follows. 

a.  The  new  magnitude  is  proportional  to  £Sq,  where  the  sum 
is  taken  over  all  the  neighbors  Q  of  P. 

b.  The  new  direction  is  basically  the  direction  to  that 

neighbor  for  which  Sq  is  greatest,  but  modified  as 
follows:  Let  the  direction  to  be  45i  degrees.  Then 
the  responses  at  Qi+1  and  Q^+g  tend  to  bias  the  direc¬ 
tion  toward  an  angle  greater  than  45i  degrees,  while  the 
responses  at  and  Q^+g  tend  to  bias  it  towards  a 

smaller  angle.  The  final  direction  estimate  is  thus 


where  Sj  is  short  for  Sq  and  S  ■  Si_i+si+Si+i+Si+3+ 
Si+4+9i+5* 

c.  The  magnitude  of  the  response  is  set  to  zero  unless  one 
of  the  following  conditions  is  met:  Let  0^02,03  be  the 
points  at  which  the  response  magnitude  is  greatest, 
second  greatest,  and  third  greatest,  respectively.  Then 
must  be  opposite  Q2»  Q2  must  be  opposite  Q3,  or  Q3 
must  be  opposite  Q^.  These  conditions  correspond  to 
the  fact  that  for  a  straight  line,  the  response  magni¬ 
tude  should  be  maximum  for  a  pair  of  opposite  directions. 

Peleg  recommends  iterating  the  above  procedure  several  times. 
For  the  current  method,  however,  a  single  application  was  used. 

As  a  result,  the  main  advantage  was  the  noise-cleaning  effect  of 
step  c.  Figure  3c  shows  the  result  of  running  the  enhancement 
process  on  the  output  in  Figure  3b. 

The  next  step  is  to  group  the  line-response  points  into  line 
segments.  This  is  done  by  means  of  a  stepwise  clustering  process. 
First,  the  points  are  clustered  into  groups  with  similar  direc¬ 
tion,  and  then,  within  each  group,  a  further  subdivision  is  made 
on  the  basis  of  the  separation  between  the  points.  Separation  is 
measured  by  rotating  the  points  so  that  their  direction  is  ver¬ 
tical,  and  measuring  the  distance  between  them.  This  distance 
must  be  less  than  a  fixed  threshold  for  the  points  to  belong  to 
the  same  line.  Because  there  is  a  fairly  large  amount  of  error 
introduced  by  the  rotation,  the  rotated  points  are  sorted  in 


order  of  increasing  distance  from  the  origin,  and  each  succes¬ 
sive  point  is  added  to  a  line  segment  if  its  distance  from  the 
previous  point  is  less  than  the  threshold.  This  provides  a 
reasonable  amount  of  laxity  in  the  grouping  process.  If  there 
are  too  few  points  in  the  line  (currently,  if  there  is  only  a 
single  point)  then  the  line  is  discarded.  Similarly,  if  the 
length  of  the  feature  in  the  image  to  which  the  line  corresponds 
is  much  greater  than  the  number  of  points  found  on  the  line 
(currently,  if  there  are  points  corresponding  to  less  than  half 
the  length  of  the  line)  the  line  is  assumed  to  be  noise,  and 
is  also  discarded.  This  prevents  widely  separated,  isolated 
points  that  coincidentally  line  up  from  being  considered  as 
valid  features. 

Each  line  in  the  low-resolution  image  corresponds  to  an 
elongated  region  in  the  original  image.  The  position  and  extent 
of  this  region  must  be  calculated,  as  well  as  a  threshold  for 
extracting  the  corresponding  linear  feature. 

At  first  sight,  it  appears  easy  to  calculate  the  position 
and  extent  of  a  region  from  the  position  and  level  in  the  pyramid 
of  the  corresponding  line.  The  endpoints  of  the  line  can  be  found 
and  mapped  into  the  ends  of  a  region  whose  extent  depends  on  the 
level  of  the  line  in  the  pyramid.  Unfortunately,  there  is  no 
guarantee  that  the  endpoints  of  the  line  are  reliable  estimates 
of  the  real  position  of  the  line.  Due  to  errors  in  the  line- 
detector  and  enhancer  (to  which  the  endpoints  are  particularly 


sensitive)  and  to  peculiarities  introduced  by  the  fixed  co¬ 
ordinate  grid  and  the  pyramid  process,  any  individual  point 
may  lie  significantly  distant  from  the  real  center  of  the  linear 
feature.  As  a  result,  the  region  calculated  may  only  partially 
overlap  the  feature  it  represents. 

An  alternative  approach  is  to  calculate  the  average  slope 
of  the  line  from  the  slopes  of  the  points,  and  to  calculate  a 
region  based  on  passing  a  line  with  this  slope  through  the 
centroid  of  the  points.  This  approach  has  the  disadvantage  that 
it  is  hard  to  calculate  the  width  and  length  of  the  region  reli¬ 
ably. 

The  method  that  was  employed  in  producing  the  examples  relies 
on  local  rather  than  global  properties  of  the  points  that  make 
up  a  line.  Points  are  treated  pairwise  along  the  line,  and  a 
series  of  regions  is  calculated,  one  for  each  successive  pair, 
based  on  the  coordinates  of  the  points.  This  insures  that  the 
local  slope  is  respected,  and  also  insures  continuity.  It  has 
the  disadvantages  of  requiring  more  computation  and  of  sometimes 
giving  a  slightly  jagged  look  to  the  output. 

A  threshold  is  calculated  for  each  line  segment  based  on 
gray-level  values  in  the  low-resolution  image.  The  threshold 
is  the  average  of  a  central  value,  derived  from  pixels  on  the 
line,  and  an  outer  value,  derived  from  pixels  on  either  side  of 
the  line.  Specifically,  the  central  value  is  the  gray-level 
of  the  point  in  the  low-resolution  image  that  gave  rise  to  the 


line-detector  response.  The  outer  value  is  the  average  of  the 
gray- level  values  at  two  points  on  either  side  of  the  line  point. 
These  two  points  are  calculated  based  on  the  direction  of  the 
line  point,  and  are  as  close  to  the  normal  to  this  direction 
as  is  allowed  by  the  co-ordinate  grid.  The  points  that  are 
chosen  are  two  pixels  away  from  the  line  point  in  the  case  of 
lines  that  are  approximately  vertical  or  horizontal,  and  one 
diagonal  pixel  away  in  other  cases.  The  threshold  value  for 
a  line  segment  is  the  average  of  the  threshold  values  of  its 
endpoints.  The  threshold  is  applied  to  the  points  in  the  ori¬ 
ginal  image  in  the  region  calculated  for  the  segment,  and  the 
results  are  written  to  the  output  image.  When  all  lines  have 
been  processed,  a  new  pyramid  level  is  constructed,  and  the  pro¬ 
cess  is  repeated. 


3 .  Examples 


Figure  4a  shows  a  small  part  of  Frederick  Municipal  Airport 
in  Maryland.  Figure  4b  shows  the  parts  of  the  image  that  were 
chosen  to  be  thresholded,  and  Figure  4c  shows  the  thresholded 
output.  It  is  clear  that  even  using  local  calculations  to  ob¬ 
tain  the  region  corresponding  to  a  line  does  not  always  give 
ideal  results. 

Figure  5  shows  three  other  airport  scenes,  including  the 
final  result  of  the  process  applied  to  Cumberland  Airport  (shown 
in  Figure  3) .  Most  of  the  linear  features  are  extracted.  It 
should  be  noted  that  thresholding  the  entire  image  using  a 
single  threshold  value  is  not  acceptable  for  these  images.  The 
results  of  such  a  process  either  do  not  contain  all  the  features, 
or  contain  large  areas  of  the  background  as  well  (Figure  6) . 

Figure  7a  shows  part  of  a  clover leaf  intersection.  There  is 
a  substantial  amount  of  curvature  in  this  feature,  and,  as 
Figure  7b  illustrates,  the  straight-line  favoring  process  almost 
completely  ignores  the  feature,  even  though  a  conventional 
thresholding  technique  would  not  be  able  to  discriminate  against 
the  curvature.  Figure  7c  shows  the  result  of  applying  the  pro¬ 
cess  without  the  enhancement  step.  Now  much  more  of  the  clover- 
leaf  is  visible,  but  at  the  cost  of  less  detail  in  the  rest  of 
the  image. 


4.  Discussion 


The  examples  in  the  previous  section  show  that  the  process 
described  is  selective  for  linear  features.  This  kind  of  selec¬ 
tivity  can  be  useful  in  its  own  right,  for  example,  when  looking 
for  roads  or  runways  in  images.  In  many  applications  such  a 
process  might  also  be  useful  as  a  first  step  in  a  more  complex 
procedure.  Suppose,  for  example,  that  an  expensive  operation 
is  to  be  performed  on  an  image,  but  need  only  be  applied  in 
restricted  regions  (e.g.  counting  cars  on  a  road  (Quam,  1978)). 

The  procedure  described  in  this  report  can  then  be  applied  to 
find  relevant  areas  in  which  to  apply  the  operation.  By  suita¬ 
bly  changing  the  feature  detector,  it  would  be  possible  to  select 
for  arbitrary  kinds  of  regions. 

The  method  is  also  potentially  useful  in  restricting  the 
search  space  for  other  procedures.  Another  approach  to  linear 
feature  extraction  that  is  being  actively  studied  is  to  use  an 
edge-extraction  technique  to  find  the  sides  of  linear  features, 
and  a  clustering  technique  to  pair  up  the  edges  that  are  extracted 
(Scher  et  al.  ,  1979,  Nevatia  and  Babu,  1978).  This  technique 
leads  to  difficulties  because  of  fragmented  edges,  and  the 
problem  of  deciding  which  edges  should  be  paired  to  form  the 
linear  features.  These  difficulties  could  be  reduced  by  first 
finding  approximations  to  the  features  using  the  method  described 
in  this  paper,  and  using  the  approximate  results  to  restrict  the 
amount  of  search  for  pairs. 


One  of  the  problems  that  became  apparent  in  applying  this 
method  was  the  large  decrease  in  resolution  between  pyramid 
levels.  Each  level  is  one-fourth  the  size  of  its  predecessor, 
and  features  tend  to  blur  significantly  between  levels.  A  fur¬ 
ther  difficulty  arises  from  the  discrete  neighborhoods  used  in 
calculating  pyramid  points.  This  tends  to  further  blur  the  image 
by  splitting  features  and  merging  parts  of  features  with  the 
background.  A  way  of  avoiding  this  problem  is  to  use  a  dif¬ 
ferent  rule  for  constructing  new  pyramid  levels.  Burt  (1980) 
gives  rules  for  constructing  pyramids  that  taper  less  sharply, 
and  whose  levels  are  constructed  using  overlapping  neighborhoods. 
It  is  intended  to  implement  some  of  his  methods  and  evaluate 
their  performance  in  this  domain. 


5 .  Conclusions 

A  method  of  segmenting  parts  of  an  image  with  specific 
properties  has  been  presented.  The  process  is  able  to  find  the 
linear  features  in  an  image,  and  segment  these  from  the  back¬ 
ground  even  in  the  presence  of  non-linear  features  with  similar 
gray- level  characteristics. 

The  method  makes  use  of  a  feature-detector  (a  line  finder) 
and  a  pyramid  of  low-resolution  images.  The  feature-detector 
finds  those  parts  of  the  image  that  have  the  desired  proper¬ 
ties.  It  is  then  possible  to  apply  further  processes  only  in 
these  restricted  parts  of  the  image.  This  results  both  in 
cheaper  procedures  and  cleaner  output. 
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Figure  1.  Mask  for  vertical  line  detection.  Point  b  is 
accepted  as  a  vertical  line  point  if  a  >  d  and  a  >  g, 
b  >  e  and  b  >  h,  c  >  f  and  c  >  i.  The  horizontal  line 
detector  is  analogous. 


Figure  2.  Mask  for  diagonal  line  detection.  For  a  45°  line 
point  b  is  accepted  if  a  >  d  and  a  >  g,  b  >  e  and  b  >  h, 
c  >  f  and  c  >  i.  The  135°  detector  is  analogous. 


Figure  3.  a.  First  pyramid  level  (Cumberland  Municipal 
Airport) .  b.  Thresholded  line-detector  output,  c.  Thresholded 
enhanced  output. 
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4.  a.  Part  of  Frederick  Airport,  b.  The  regions 
for  thresholding,  c.  The  thresholded  output. 


Figure  5.  Three  airfields  and  the  results  of  applying  the 
linear-feature  detector  to  them. 
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Figure  6.  a.  A  global  threshold  producing  results  similar 
to  those  in  Figure  5  is  obtainable  only  at  the  cost  of  missing 
a  vertical  linear  feature,  b.  When  all  the  linear  features  are 
extracted,  so  is  a  large  amount  of  the  background. 
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Figure  7.  a.  A  cloverleaf  intersection,  b.  The  result  of 
applying  the  detector  to  (a).c.  The  result  of  applying  the 
detector  without  line  enhancement. 
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Image  processing 
Pattern  recognition 
Linear  features 
Segmentation 
Pyramids 
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A  method  is  described  of  extracting  linear  features  from  images. 
The  approach  is  to  construct  a  series  of  lower-resolution  versions 
of  the  original  image  (a  pyramid) ,  and  to  look  for  lines  in  these 
images.  A  line  in  a  low-resolution  image  corresponds  to  a  thicker 
linear  feature  in  a  high-resolution  image.  The  position  and  extent 
of  this  linear  feature  is  calculated  from  the  low- resolution  image, 
and  a  threshold  is  found  which,  when  applied  in  the  neighborhood  of 
the  feature  in  the  high-resolution  image,  segments  the  linear 
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feature  from  its  background.  Advantages  of  tha  method  ara  that 
only  tha  parti  of  tha  image  in  tha  neighborhood  of  linear 
feature!  need  be  thresholded,  and  that  different  thresholds  may 
be  used  to  extract  the  various  linear  features  in  the  image. 
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